In recent years, there has been a rapid increase in demand for delivery of real-time applications and services in computer networks, including Pseudo-Wire Emulation (PWE), Voice over IP (VoIP), video conferencing, and broadcast, multicast and manycast streaming services such as H.261, H.323, and IPTV. These real-time services may require highly accurate timing to ensure high service quality. For example, it is desirable to eliminate data loss due to clock mismatch between the source and the destination. This can be done by providing a highly accurate timing reference at the source and at the destination, such as a Global Positioning System (GPS) reference or a lower quality oscillator such as a Stratum 2 rubidium oscillator, where the specification for Stratum 2 clock quality is given in Telcordia GR-1244-CORE. However, at the same time it is desirable to reduce the substantial cost resulting from per-node deployment of these timing references.
FIG. 1 illustrates a network architecture including a single time server 100 that provides timing information to client devices 106A-106N. The time server 100 may be a Network Time Protocol (NTP) server, and the timing information may be contained in timing packets 101 that traverse network 110 to a time relay server 104. These timing packets 101 traverse switching devices 102A-102N in network 110. At each switching device 102, the timing packets 101 are multiplexed with data packets 120. Each switching device 102 may use conventional queuing such as store-and-forward queuing. The time relay server 104 may synchronize to the timing information from the time server 100 contained in timing packets 101, and may generate and transmit timing information contained in timing packets 105A-105N to the clients 106A-106N. The clients 106 may synchronize to the timing packets 105 received from the time relay server 104.
One of the important factors that limits the timing accuracy is variations in network delay, known as network jitter, experienced by the timing packets 101 due to multiplexing with the data packets 120. The minimization of network jitter, and correspondingly the tight bounding of network delay experienced by the timing packets 101, enhances the timing accuracy of timing distribution protocols, and improves the quality of real-time network applications and services. The timing accuracy of the timing distribution protocol NTPv4 over the public Internet may be on the order of 10 milliseconds; in local area networks, the timing accuracy of NTPv4 may be better, on the order of hundreds of microseconds. In practice an incoming jitter with a standard deviation of 100 nanoseconds on a Stratum 1 referenced clock can be filtered to provide Stratum 2 quality timing distribution, which is desirable to remove the need for a GPS reference or a Stratum 2 rubidium oscillator at each client 106. However, the timing accuracy of conventional NTPv4 implementations appear to be far from what is needed for distribution of Stratum 2 quality timing.
In packet networks the delays are primarily the station to station transmission delay over the physical media and transit hop delay. A packet, such as timing packet 101, experiences transmission delay between transit stations such as switching devices 102, and transit hop delays, including media access control (MAC) delay and queuing delay, at each switching device 102. End-to-end delay is the total delay that the timing packet 101 experiences from the source, such as time server 100, to the destination, such as time relay 104. The end-to-end delay includes, in addition to all the delays experienced per transit hop, MAC and queuing delays at the source and destination nodes.
Transmission delay is the delay due to the distance the signal travels at the speed of light over the associated physical media. In normal operation, transmission delay is slowly varying due to thermal and diurnal effects. Delay variation that is slowly varying is known as wander. Transmission wander due to thermal and diurnal effects in most networks is generally small, on the order of 100 nanoseconds over 5000 kilometers of fiber, and can be tracked and easily compensated.
FIG. 2 illustrates the components of media access control delay 200 and network queuing delay 202, and the accumulation of these components as a timing packet 101 traverses network switches 102. The media access control delay 200 includes media access (MA) minimum delay 210, MA jitter 212, and synchronization jitter 214. The network queuing delay 202 includes network queuing (NQ) minimum delay 216 and NQ jitter 218. A packet transiting network switch 102, such as a timing packet 101, may experience as little delay as (MA minimum delay 210+NQ minimum delay 216) before transmission at the earliest possible packet transmission time 230. In this case, all of the components of jitter are zero. Generally a packet transiting network switch 102 may experience a delay of (MA minimum delay 210+MA jitter 212+Synchronization jitter 214+NQ minimum delay 216+NQ jitter 218) before transmission at the actual packet transmission time 232. The minimum transit delay accumulated through N network switches 102 may be N*(MA minimum delay 210+NQ minimum delay 216). The jitter through N network switches 102 may be uniformly distributed with standard deviation (Max(MA jitter 212)+Max (Synchronization jitter 214)+Max(NQ jitter 218))*(N/12)1/2.
The MA delay 200 is due to the protocols that schedule transmissions on the physical media, and depends upon bit transmission rate, inter-packet gaps, packet fragmentation, inverse multiplexing and similar effects. In the case of a point-to-point full duplex Ethernet link operating at 100 Mbps, the MA delay 200 is primarily due to the 96-bit Inter-Packet Gap (IPG) between the start of a packet transmitted from a MAC device within network switch 102 and the end of the preceding packet, plus some MA minimum delay 210 that includes minimum MAC processing delay. Traditional queuing approaches, such as store-and-forward transit queuing, allow a packet transiting network switch 102, such as a timing packet 101, to be unpredictably delayed. This is because the full 96-bit IPG or a portion thereof may need to be inserted on the outgoing link after the end of transmission of a packet inserted at network switch 102, such as a data packet 120. At 100 Mbps, this MA jitter 212 due to IPG is 0 to 96 bits, or 0 to 960 nanoseconds. Since each network device along the path from the time server 100 to the clients 106 can experience MA jitter 212, the standard deviation of the accumulated end-to-end jitter across multiple network switches 102 due to MA jitter 212 alone can far exceed the 100 nanosecond target for Stratum 2 timing distribution. For example, the standard deviation of the accumulated jitter after 100 network switches 102 due solely to MA jitter 212 is 960 nanoseconds*(100/12)1/2=2.77 microseconds. This means that traditional store-and-forward transit queuing may not be acceptable for highly accurate transmission of Stratum time and frequency in a packet network.
A secondary source of variation in the MAC delay 200 is due to synchronization required to prevent flip-flop multi-stability when passing packet data across clock domains. The uncertainty for each synchronization stage may be one clock cycle of the synchronizing clock. For example, after the timing packet 101 is received at a network switch 102, the timing packet 101 may be transmitted on the next rising edge of the transmit MAC interface clock at the network switch 102. In this case, the synchronization jitter 214 may be bounded by one clock cycle of the transmit MAC interface clock. In 100 Mbps Ethernet the underlying rate of the transmit MAC interface clock is 100 MHz. The resulting synchronization jitter is uniformly distributed and bounded between 0 to 10 nanoseconds per network switch 102. The standard deviation of the accumulated jitter after 100 network switches 102 due solely to synchronization jitter 214 is 10 nanoseconds*(100/12)1/2=28.86 nanoseconds. This means that the standard deviation of the accumulated synchronization jitter 214, by itself, appears to be within the 100 nanosecond target for distribution of Stratum 2 quality timing.
Network queuing delay 202 is perhaps the largest source of delay and jitter in a network. Network queuing can be caused by output port contention arising from packets arriving from a multiplicity of input ports with the same desired output port, such as a transit timing packet 101 and an incoming data packet 120 multiplexed at network switch 102. Only a single packet can egress the network switch 102 at any given time, so any other contending packets may be queued or dropped. The magnitude of the NQ minimum delay 216 and the NQ jitter 218 is shown for a representative example. Implementations of standard Ethernet have a maximum packet size, or Maximum Transfer Unit (MTU), of 1518 bytes. Not including the IPG transmission time, the transmission time for a 1518 byte packet at 100 Mbps may be 121.44 microseconds. The transmission time for a minimum size 64 byte packet may be 5.12 microseconds. In a 100 Mbps Ethernet store and forward network, the NQ minimum delay 216 may be 5.12 microseconds, such as for the queuing time of a 64 byte timing packet 101 with no packet directly in front of it or multiplexed with it. Assuming that the timing packet 101 has strict priority over all data packets 120, the maximum NQ jitter 218 may be 121.44 microseconds, such as for the queuing time of a 64 byte timing packet 101 waiting for a full 1518 byte data packet 120 to be multiplexed in front of it. (Note that delays far in excess of 1 millisecond may occur otherwise.) This NQ jitter 218 may happen at every network switch 102 traversed by the timing packet 101 as it is forwarded from the time server 100 to the time relay server 104. The standard deviation of the accumulated jitter after 100 network switches 102 due solely to NQ jitter 218 may be at least 121.44 microseconds*(100/12)1/2=350.57 microseconds. (Note that in real networks, the standard deviation of the accumulated NQ jitter 218 may be much higher than this due to the self-similar nature of many network traffic patterns.) This means that the standard deviation of the accumulated NQ jitter 218, by itself, appears to be far in excess of the 100 nanosecond target for distribution of Stratum 2 quality timing.
The above discussion shows that the standard deviation of the jitter accumulated by timing packets 101 that traverse many network switches 102 using conventional store-and-forward queuing appears to be far higher than the target standard deviation that can be filtered to provide highly accurate network timing, such as 100 nanoseconds for Stratum 2 timing. Moreover, the statistics of this accumulated jitter may depend heavily on highly self-similar traffic patterns and thus may be extremely complex to filter. To address this shortcoming, it would be desirable to provide a mechanism for forwarding packets at each network switch 102 that eliminates most or all of the MA jitter 212 and the NQ jitter 218, which appear to be, by orders of magnitude, the primary sources of jitter accumulation impacting the transiting of timing packets 101. This may enable timing packets 101 to be distributed with a standard deviation of accumulated jitter of less than 100 microseconds. The elimination or reduction of the NQ jitter 218 may also substantially simplify any filtering that may need to be performed.